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ABSTRACT 



This paper presents an overview of a plenary panel entitled 
"Measuring Outcomes and Setting Standards in Languages Education." "Outcomes" 
implies what learners take away from their course, while the term "standards" 
seems commonly to have at least two senses: first it may imply some yardstick 
or framework against which learner performance, the content of tests and 
examinations, or the goals of the courses may be measured; second, it implies 
a "framework of reference." This brief introduction to the panel discussion 
introduces some of the broad issues touching on such topics as quality and 
standards in language education, setting goals and objectives, course-based 
versus general proficiency outcomes, assessing cultural goals and cross 
cultural attitudes, teacher quality and supply, relevance of vocational 
language competencies, and exit assessment and reporting. This paper seeks to 
raise some of the general issues that may be elaborated upon by other 
panelists and in subsequent discussion. (Contains 10 references.) (KFT) 
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Measuring Outcomes and Setting Standards: A Brief 

Overview 

D. E. Ingram 

Director, Centre for Applied Linguistics and Languages, 
Griffith University, Brisbane 



Abstract 

This brief introduction to the panel discussion will introduce some of the broad issues 
touching on such topics as quality and standards in language education, setting goals 
and objectives, course-based vs. general proficiency outcomes, assessing cultural 
goals and cross-cultural attitudes, teacher quality and supply, relevance of vocational 
language competencies, and exit assessment and reporting. This paper will do no 
more than raise some of the general issues that may be elaborated upon by other 
panellists and in the subsequent discussion. 
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Measuring Outcomes and Setting Standards: A Brief 

Overview 

D. E. Ingram 

Director, Centre for Applied Linguistics and Languages, 
Griffith University, Brisbane 



I INTRODUCTION 

On the final day of the Joint National Conference, held in Adelaide in July 1999, a 
, plenary panel was devoted to the topic Measuring Outcomes and Setting Standards in 
Languages Education. The session took the form of four short presentations by (in 
order of presentation) David Ingram, Angela Scarino, Lyle Bachman, and Penny 
McKay. The presentations were followed by discussions in which the panel 
responded to questions and comments from the audience. The session seemed to be 
well received and a considerable number of delegates expressed a wish for the papers 
to be re-produced in Babel. The papers are presented here in the sequence in which 
they occurred at the conference. 



II THE PANELLISTS 

The first paper was presented by the present writer. David Ingram is Professor of 
Applied Linguistics and Director of the Centre for Applied Linguistics and Languages 
in Griffith University in Brisbane. He was President of AFMLTA for fourteen years 
from 1982 to 1996. In his presentation, he sought to provide a very brief overview of 
the topic. The second speaker was Angela Scarino, President of AFMLTA since 1996 
and a Head of Department in the University of South Australia. Angela probably first 
became well known across Australia when, throughout the latter half of the 1980s, she 
was Director of the Australian Language Levels Project. Angela’s paper was entitled 
“The Concept of Standards” and she discussed the nature of “standards”, how 
standards are set, and the roles of teachers in the process. The third panellist was the 
eminent. Dr Lyle Bachman from the University of California in Los Angeles. Lyle 
has filled a world-leading role in language testing. He spoke on “Considerations in 
Measuring Learner Outcomes of Language Learning Programs” and considered such 
issues as purposes, interpeting measures used, and the relevance of stakeholders, 
programs and learners. The final panellist was Dr Penny McKay, a Senior Lecturer in 
the education faculty of the Queensland University of Technology, and, like Angela, 
had first become well known for her involvement on the Australian Language Levels 
Project, giving particular attention to the ESL area. Penny spoke on political aspects 
of standards setting and the intent of governments in introducing so-called standards, 
making reference to Australia, Britain and the United States. She also referred to the 
tension between pedagogic and administrative purposes. 



IH THE NOTION OF STANDARDS 
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The plenary panel was entitled Measuring Outcomes and Setting Standards in 
Languages Education. “Outcomes” implies what learners take away from their 
course; while the term “standards” seems commonly to have at least two senses: first, 
it may imply some yardstick or framework against which learner performance, the 
content of tests and examinations, or the goals of courses may be measured. It was 
used in this sense to refer to the level of an examination in the Council of Europe’s 
1973 publication on a European unit/credit system for adult modem language 
learning, of which John Trim said: 

The multidimensional operational classification provides a framework for the 
analysis of the content and standard of existing and new examinations and 
tests and their placement in a system of equivalences^ Trim 1973: 23] 

4 

Later Trim refers to “the hypothetical standard of the examination” in respect of such 
things as grammatical accuracy, spelling and pronunciation, vocabulary, 
appropriateness, richness of expression, range of comprehension, and fluency, [cf. 
Trim 1973: 24] 

More recently, the Common European Framework [Education Committee, Council 
for Cultural Cooperation, Council of Europe 1996] seems to avoid the use of the term 
“standards”, speaking of a “framework of reference” and “scaling” to describe a 
systematic, comprehensive and coherent framework within which, amongst other 
things, 



Learners, teachers, course designers, examining bodies and educational 
administrators [can] situate and coordinate their efforts ... [Education 
Committee, Council for Cultural Cooperation, Council of Europe 1996: 2] 

It is a small but important step from this to the second basic sense in which 
“standards” is commonly used, where it carries some notion of what students ought to 
achieve at one level or another or for some purpose. So, for instance, the Common 
European Framework (or the scales within it) becomes a “standard” (or standards) 
when used to specify what learners should achieve at various levels, what skills are 
required for entry to, for example, particular professions, or the content to be included 
in courses or examinations at particular levels. The setting of curriculum parameters 
and the specification of desired achievement levels inherent in some uses of the term 
“standards” is well caught in Victoria’s newly revised “Curriculum and Standards 
Framework”, where the Framework is described in these terms: 

The Curriculum and Standards Framework (CSF) is the basis upon which 
Victorian schools plan and deliver curriculum and monitor student 
achievements. It describes what students should know and be able to do at 
various stages during the years Prep to Year 10. It provides sufficient detail 
for schools and the community to be clear about the major elements of 
curriculum and the expected outcomes. [Draft document received from the 
Victorian Board of Studies, dated 15 June, 1999. Mimeograph. (Present 
writer’s emphases)] 
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Used in such ways, the term “standards” often acquires some administrative, 
political, and, at times, even moralistic value, largely dependent on the purposes for 
which the standards are set. 

The notion of a standard is quite defensible and serves valuable purposes if it is set 
with some rational justification or for some valid purpose such as providing a 
curriculum framework, assisting in the development of syllabuses and work 
programmes, or in interpreting and measuring student achievement in some valid, 
appropriate, and meaningful way. So, for example, a proficiency level may become a 
“standard” that learners must reach when it is specified for university entry or to meet 
a professional registration requirement. Such a “standard” may be objectively 
justifiable if, for instance, the proficiency specified is related to the sort of language 
behaviour that is required in undertaking study in the particular programme of study 
the intending student is entering or if it relates to the sort of language behaviour 
routinely undertaken in carrying out the practical duties of a vocation whose 
performance “standards” a Vocational Registration Board has been established to 
regulate; many Australian and British universities, for example, set IELTS 6 or 6.5 as 
the standard to be reached in English before overseas students will be admitted to 
their courses and increasing numbers of vocational registration boards in New 
Zealand, Britain and Australia are acting similarly though with the standard usually 
being set at IELTS 7 or higher. 

However, it is difficult to justify the notion of a “standard” if it is arbitrary, not 
obviously related to or explicable in terms of real language use requirements, or if it is 
set as, for example, an arbitrary score on some arbitrarily chosen test to serve some 
administrative or political expediency. Some frameworks of reference exist as 
specifications of competencies that claim to be a statement of the micro- and macro- 
tasks that persons have to carry out in some vocation or in some other activity; in this 
they may provide a quite reasonable statement of observable language behaviour and 
language requirements to undertake the activity but they can become quite 
inappropriate if used as “standards” for purposes for which they were not designed or 
for which the fundamental requirements are quite different. So, for example, a set of 
vocational competency specifications that may quite accurately identify the skills and 
tasks required in the workplace need not be related to how language develops and is 
quite inappropriate for use in setting levels or “standards” in a sequence of language 
programmes or for reporting on learning outcomes other than in a flat non- 
developmental sense to state what vocational tasks the learners can carry out. 
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In the moralistic or political sense, the term “standards” has frequently been abused. 
Too often, the term becomes quite empty of “denotative meaning”, is used largely 
emotively with, at best, commonly recognised connotative meanings vaguely related 
to commonly held values, and is used for largely political purposes relating, at best, to 
some general notion of what people feel students should be achieving. Cynically, 
politicians seem to like to speak of “standards” so as to show their supposed concern 
for education and to make electors believe that they are “doing something” about 
education or “would do something about education if only they were in power” with 
little notion of what levels students are actually reaching, what levels it is reasonable 
to expect them to reach, or, least of all, what resources education requires if it is to be 
changed or its “quality” raised. In other instances where the use of some notion of 
“standards” is quite appropriate and is in the interest of learners and of other 



6 



6 



stakeholders, the level set as the “standard” may be arbitrary and only vaguely related, 
if at all, to the nature of language or of the language skills required for some purpose. 
So, for instance, in the case of university entry by overseas students, standards are too 
often determined by the gut-feeling of university admissions officers or avaricious 
marketing persons without consulting with or accepting the advice of anyone with 
knowledge of the tests used or what their results mean: so what may, in fact, be a 
perfectly reasonable means of identifying language performance requirements and 
measuring and stating learner skills loses its reasonableness because the “standards” 
are set for other reasons such as administrative expediency or greed for overseas 
student fees irrespective of the students’ ability to perform academically. Equally 
reprehensible misuse of standards setting is seen where instruments appropriately 
used to set defensible “standards” in one area are used inappropriately in other areas 
because it is administratively convenient to do so. So, for instance, the recent 
decision by the Australian Immigration Minister to abandon the ACCESS test, which 
was designed to measure the social or vocational language skills of applicants for 
migration to Australia, in favour of the IELTS test, which was designed specifically to 
measure the English language skills required for academic and training purposes in 
English-speaking institutions seems to have been made for financial and 
administrative reasons with no regard to the appropriateness of the tests for setting 
standards in the respective areas. That practice is, however, marginally less 
reprehensible (since both tests are designed for and used with spearkers of English as 
a second language) than the practice of some vocational registration boards overseas 
which, apparently for reasons of administrative expediency, require IELTS (a test 
designed for second language learners) as a convenient, even if inappropriate, 
gatekeeper for overseas-trained native English-speaking applicants. 

If used simplistically as a framework of reference against which to evaluate language 
learning or the worth of other educational programmes, so-called “standards”, 
especially if arbitrarily determined, may over-simplify the educational process, ignore 
the variety of factors that determine actual learning outcomes, and have a restrictive 
washback effect on learning and teaching. Thus, apart from their educational 
irrelevance, one of the overwhelming disadvantages of simplistic standards, especially 
of the political or moralistic kind, is that they become domineering and take the focus 
away from issues (such as cross-cultural attitudes and cultural understanding) that are 
educationally valuable but do not lend themselves to precise specification, to easy or 
reliable measurement, and hence to ready standards setting. As language teachers 
know, the frequent outcries of low “standards” in the so-called “basics” of reading, 
writing and calculation are generally unrelated to any specification of what the 
desirable “standards” might be or of what students are actually achieving or could 
achieve but they often have a powerful effect on the curriculum, narrowing it, and, not 
least, reducing support for other activities such as the teaching and learning of foreign 
languages, the resources available for them, and the funding available for supporting 
'research and development: the deplorable situation that has occurred in language 
education in Australia since 1996 well illustrates the dangers of the emotive and 
political use of so-called “standards”. 

In brief, the notion of “standards” has both values and inherent dangers: “standards” 
can be of value when they relate to some meaningful aspect of language or to 
something inherent in the nature of language learning and its systematic development. 
In this, they may provide useful frameworks within which needs may be specified for 
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the purposes of language and language education policy development, and in which 
curriculum development, learner assessment and programme evaluation may take 
place. However, they are inappropriate and generally positively harmful if they have 
little relationship to the nature of language, language behaviour and language learning 
or if they are set and imposed without being properly matched against desirable 
purposes. 



IV OUTCOMES AND THEIR MEASUREMENT 

“Outcomes” refer to what learners take away from the course: they may match the 
formally stated goals and objectives but generally differ to a greater or lesser degree 
from them and invariably include things such as attitudes to language teachers that 
may not be included in goals but which may loom large in determining, for example, 
whether students continue with language study or not. Outcomes may be expressed in 
terms of some specified framework of reference such as, for language skills, a 
proficiency scale or language competencies, i.e., in terms of some set of “standards”, 
but often they are not . Outcomes from a language course are diverse and are not just 
related to language performance or knowledge of the elements of the language, they 
are measured in different ways, some lend themselves to being measured but some 
don’t, some arguably should be measured as part of the assessment of student 
performance but others, even though they might be important as goals, do not lend 
themselves to formal assessment and are better not included in a formal assessment 
programme. 

In the present writer’s views, the three central goals and, hence, the three central 
outcomes of any language programme (though by no means the only outcomes) are 

• the attainment Of language proficiency, 

• the attainment of cultural knowledge and understanding, and 

• the attainment of more favourable cross-cultural attitudes. 

Comprehensive frameworks of reference such as the Common European Framework 
attempt to provide very comprehensive, systematic and detailed specifications for 
such goals and provide sets of criteria against which, potentially, outcomes can be 
measured. Others, such as the National Standards for Modern Foreign Languages of 
the Languages Lead Body in Britain or the so-called National Reporting System in 
Australia focus on particular aspects of language use and language learning, viz., 

' vocational competencies, essentially the language tasks, knowledge, and attitudes 

required for the workplace. 

In this very brief overview, outcomes related to the three central goals can be seen to 
fall into four categories: 

IV.l Course-related Outcomes and Measurement: In any course, teachers, 
learners and funding providers are likely to be interested in the extent to which course 
content has been assimilated and specified skills and attitudes developed. These may 
include knowledge of the particular language elements included in the course (the 
vocabulary, grammar, functions, and so on), ability to perform specified language 
tasks, or familiarity with the bits of cultural knowledge encountered. The form that 
O 
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measurement takes will differ according to the nature of those elements but will 
generally be closely related to the sorts of teaching and learning activities that have 
gone on through the course. In addition, however, overall course assessment may, 
legitimately, be concerned with how well the students have performed as students 
e.g., how much progress they have made, whether they have been attentive and 
cooperative, whether they have done their homework and got their assignments in on 
time and with good presentation, and so on. 

For some purposes, including, for instance, selection of students for subsequent study, 
course-related assessment including such things as continuous assessment based on 
assignments, projects, and in-class performance may be legitimate ways to measure an 
outcome such as “student performance” but it is important to realise what is being 
measured here and not to confuse it with, for instance, the students’ exit language 
proficiency, as is done when school grades based on continuous assessment are 
assumed to indicate students’ level of exit language skills. Thus, most end-of-Year 12 
and tertiary entry results (under various names such as HSC, OPs, Matriculation, A- 
Level, and so on) have at least as much to do with measuring and stating performance 
as a student as they have with stating students’ language skills. 

In brief, course-based outcomes and measurement are legitimate for certain purposes 
but one needs to be clear about their nature and purpose and interpret and use the 
measurement outcomes appropriately. 

IV.2 Language Skills: Most language teachers believe that the outcomes they most 
seek are improved language skills. Traditionally these have been assessed in terms of 
formal knowledge of elements of the language and many tests, including major 
international tests, still have a strong focus on knowledge of grammar, vocabulary and 
other elements of the language. Since the emergence of communicative language 
teaching, however, the focus has been on learners’ ability to carry out communicative 
language tasks or the more abstract concept of language proficiency. Language 
proficiency is commonly expressed in terms of language proficiency scales such as 
the ISLPR [Ingram and Wylie 1979/1995] or the IELTS Bandscales, which attempt to 
describe the language behaviour observable as learners develop their language from 
zero proficiency to what some scales term “native-like proficiency”. Language 
proficiency may be described in general terms applicable to a user’s performance in a 
wide variety of situations, i.e. “general proficiency”, or it may be described in more 
specific contexts or for more specific purposes such as academic or specified 
vocational purposes. So, for instance, the ISLPR exists as scales of general 
proficiency in a variety of languages but also there are versions in English for 
Academic Purposes, English for Engineering, a generic version for LOTE teachers, 
and a version for teachers of Indonesian. One common, even if not inevitable, 
characteristic of proficiency scales is their developmental nature, i.e., they don’t just 
describe language behaviour but also attempt to show how language develops from 
one level to another, some from zero proficiency to native-like. This feature enables 
proficiency scales to be used to specify goals and objectives but also to provide an 
overall framework within which curricula and a series of syllabuses may be 
developed. 

Language skills have also been described in terms of the micro- and macro-tasks or 
“competencies” that learners can carry out or that they need for various purposes, 




9 



9 



most commonly vocational purposes. So, for instance, the National Standards for 
Modern Foreign Languages developed by the Languages Lead Body in Britain or the 
National Reporting System developed by Sharon Coates and her colleagues in 
Australia purport to describe the competencies required to use language in the 
workplace. Clearly such competency specifications are of value in identifying the 
tasks that language users need for vocational purposes, they can be used to assess 
whether learners have the skills required for employment in particular activities, they 
can be used to assist industry to identify and state what skills are required to carry out 
the tasks for which they wish to recruit people, and they can have useful input to the 
development of curricula and syllabuses intended to develop vocationally useful 
language skills. However, vocational competency specifications focus on workplace 
tasks, they don’t need to be developmental in their organisation, nor do they need to 
take account of the different learning requirements of students in first or second 
language learning programmes, and hence they are not so appropriate as an overall 
curriculum framework within which a developmental sequence of language 
programmes can be developed or learner outcomes developmentally specified. 

IV.3 Cultural Knowledge and Understanding: Like language skills, the 
development of cultural knowledge and understanding has traditionally been seen as 
one of the essential goals of language teaching. Indeed, since the culture is the 
meaning system that underlies the language, it is impossible to conceive of 
meaningful language use and communication-focussed language learning without the 
development of cultural knowledge and understanding. For these reasons, there has 
been a strong tendency in recent years, at least in Queensland, for culture not to be 
taught or assessed separately from language skills. This was, initially, a defensible 
reaction against the simplistic form of culture teaching that often occurred decades 
ago when teachers dictated lists of monuments and historical or geographical facts, 
students memorised them, and “outcomes measurement” took the form of short 
factual questionnaires. However, a recent survey conducted by the present writer in 
Brisbane schools [Ingram 1999, 1999a, 1999b] suggests that an equally undesirable 
consequence of this has been a neglect of culture teaching by many teachers. It is 
difficult to recommend an easy compromise but, rather than re-introduce to end-of- 
course assessment questionnaires about culture facts, it is probably better for teachers 
to ensure that there are many activities throughout a course that lead to development 
of cultural knowledge and understanding and for the acquisition of that knowledge 
and understanding to continue to be seen as integral to language performance and to 
be measured or assessed in conjunction with language performance. In addition, it 
may be possible to develop scales that describe levels of cultural knowledge and 
under standing just as there are scales that describe levels of language proficiency but, 
as yet, there seem not to be any very adequate scales of this sort and some major 
scales that initially included culture sub-scales have dropped them. Even so 
comprehensive an instrument as the Common European Framework implies cultural 
competence rather than elaborates on it and, in the one page that discusses cultural 
competence in a volume of over 200 pages, acknowledges that 

All aspects of socio-cultural competence are .. very difficult to scale ... 

[Education Committee, Council for Cultural Cooperation, Council of Europe 

1996: 133] 
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Again, as noted earlier, the fact that it is difficult to measure cultural knowledge and 
understanding does not invalidate it as a desirable outcome nor does it reduce the 
importance of including the development of cultural knowledge and understanding in 
syllabuses, work programmes and teaching activities. 

IV.4 Attitudes: The development of more favourable cross-cultural attitudes is a 
central goal of language teaching and is one of the outcomes that most language 
teachers hope that learners gain from a course [cf. Ingram 1978, 1999, 1999a, 1999b]. 
In addition, language teaching and learning, like all other activities in education and 
beyond, inevitably entail the development of other attitudes: attitudes to the subject, 
attitudes to language learning, attitudes to the teacher, attitudes to self-directed or 
teacher dominated learning, attitudes to the school, and so on. By no means all of 
these are likely to be specified in any statement of goals and objectives for a language 
course but, specified or not, most of them are inevitably inherent in the educational 
process and are part of the outcomes from any course. However, the formal 
measurement of any of these, including even the attainment of more positive cross- 
cultural attitudes, is difficult, time-consuming and notoriously unreliable. The basic 
cause of the unreliability, quite apart from the difficulty of devising suitable, valid and 
reliable measurement instruments, is that learners are likely to respond to questions 
about attitudes in the way that they expect the teacher or examiner would want them 
to respond and, indeed, in outcomes measurement for some significant purpose, such 
as for selection to the next stage of education, they would be foolish to do otherwise. 

It may be possible for teachers to observe learners’ attitudes throughout a course, to 
gain an impression of them, and to assess them subjectively over time using 
continuous assessment. It is also possible for researchers to develop attitude 
questionnaires that have reasonable levels of validity and reliability [Ingram 1999, 
1999a, 1999b] even though the problem of having learners respond as they think the 
test writer wants them to respond is probably insuperable. However, such tests tend 
to be long, complex, and difficult to construct and, all in all, it would seem to be 
neither practical nor appropriate for formal measurement of attitudes, even of cross- 
cultural attitudes, to be included in formal end-of-course or outcomes measurement 
but this does not reduce the importance of deliberately including the development of 
more positive cross-cultural attitudes in the goals and objectives of language courses 
and of ensuring that the methodology adopted is such as to make the development of 
more positive attitudes (both cross-cultural and other attitudes) more likely to be 
achieved. 



i 

V CONCLUSION 

This paper attempted to provide only a very brief and cursory overview of what is a , 
highly complex and often quite contentious issue, the measurement of outcomes and 
the identification and setting of standards in language programmes. The other 
panellists, whose presentations follow, selected a range of different aspects of the 
issue to discuss. 
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